The approaches described in this section are approaches that could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
Some digital cameras are able to produce digital images at varying desired levels of zoom without the need for expensive and failure-prone mechanisms that physically move the lens closer to or farther away from the camera's sensor array; the lens may remain stationary. An example technique for achieving optical zoom using a stationary lens system that captures a distorted image is described in PCT Patent Application PCT/IB2007/004278, Publication No. WO 2008/152441 A2, filed on Sep. 14, 2007, titled “IMAGING SYSTEM FOR IMPROVED IMAGE QUALITY AND ASSOCIATED METHODS,” the contents of which are incorporated herein by reference.
Generally speaking, the stationary-lens zoom feature may be accomplished through the use of a system of lenses that capture a radially distorted image, in which the central portions of the distorted image are more magnified (and therefore more detailed) than are the portions along the distorted image's periphery. The radially distorted image may be, but does not necessarily need to be, an image that is produced via a fisheye-like distortion. Whereas normal high-quality lenses might naturally produce a maximum distortion of less than 2% in portions of the captured image, the lens system used in the distorted zoom technique may intentionally produce a distortion of over 40% in portions of the captured image.
In some implementations, the sensor array onto which the distorted image is captured is at a resolution that is higher than the resolution at which any corresponding undistorted image reconstructed from the distorted image will be. For example, the sensor array may capture an 8 megapixel distorted image, while any corresponding undistorted image reconstructed from the information captured by the sensor array might be only a 5 megapixel image, regardless of the desired level of zoom. Although image distortion may be decoupled from an oversized sensor, and each alone can be used to gain some zoom compatibility, the two may be used in concert to produce a synergistic effect.
Because more detailed image information is captured toward the sensor array's center due to the magnification attending the radial distortion, a zoom-in feature can be implemented by passing most of the highly magnified, highly detailed image information close to the sensor array's center on to the reconstructed image while excluding, from the reconstructed image, less magnified, less detailed image information from the sensor array's periphery. In a highly zoomed-in output image (resembling an image that might be produced using a telephoto lens), the less magnified, less detailed image information at the periphery of the sensor array will be beyond the scope of the zoomed-in image's view frame anyway.
However, if a completely zoomed-out image (resembling an image that might be produced by a wide-angle lens) is desired, then relatively fewer values of the sensors in the higher-resolution sensor array may be used as values for corresponding pixels in the lower-resolution reconstructed image. Relatively few sensor values from the central parts of the sensor array will influence the reconstructed image's central pixel values, thereby reducing magnification and detail at the center of the reconstructed image to the level of magnification and detail found at the periphery of the sensor array.
Using the distorted zoom technique described above, very little pixel estimation is required in order to generate a zoomed-in image. While digital zoom techniques seek to estimate missing output pixel values based on the values of the pixels adjacent to those pixels, which often results in imprecision, the distorted zoom technique described above optically captures all or almost all of the pixel information that will be required for a zoomed-in image. Again, this lack of need for much pixel estimation is due to the increased magnification of the central portions of the lens system's subject on the sensor array, at a level of magnification and detail that is already at or nearly at the level of magnification and detail that will be needed in the most maximally zoomed-in reconstructed image. Essentially, the radial distortion and higher-resolution sensor array work together to “oversample” the information located towards the center of the lens system's subject so that optical zoom, rather than digital zoom, can be achieved.
Because the sensor array captures a distorted image, image processing is performed during image reconstruction so that the resulting reconstructed image is not distorted. One technique for reconstructing an undistorted image is known as “backward interpolation”. Under the backward interpolation approach, pixel values read out of the sensor array are temporarily stored in a set of input line buffers. Since the input line buffers are used to store pixel values from the sensor array, the number of cells in each input line buffer is equal to the number of columns in the sensor array. Because the number of columns in the higher-resolution sensor array is greater than the number of pixels in each row of the reconstructed image, there are more cells within each input line buffer than there will be pixels in each row of the finally reconstructed image. Initially, the input line buffers are filled with pixel values from the uppermost rows of the sensor array. For example, if there are N input line buffers, then, initially, the input line buffers are filled with pixel values from the topmost N rows of the sensor array—one input line buffer per sensor array row. Typically, due to the expense of line buffers, the quantity of line buffers in the set of input line buffers will be less than the quantity of rows in the sensor array. Typically, effort is made to restrict the quantity of line buffers in the set of input line buffers to the minimum quantity that may be required to perform the backward interpolation technique.
Under the backward interpolation technique, pixel rows of the reconstructed image are filled in raster order, one pixel row at a time, starting at the leftmost pixel of the pixel row currently being filled (initially, the topmost pixel row in the output image) and proceeding to the rightmost pixel of that pixel row before continuing on to the next-lower pixel row in the output image. For each pixel (referred to in this discussion as an output pixel) in a row of the reconstructed image, a backward mapping is performed to map that output pixel to a corresponding pixel in the sensor array (referred to below as the input pixel). The pixel value for this input pixel, which is stored in a corresponding input line buffer cell, is used (perhaps in conjunction with the pixel values of some neighboring pixels around the input pixel) to determine the pixel value of the output pixel. The reconstructed image's pixels are filled in this manner, one-by-one, row-by-row. During this process, if it can be ascertained with certainty that a particular input line buffer's data no longer influences the pixel values of any pixels of the reconstructed image's pixel row currently being filled, then that particular input line buffer's data is replaced with pixel values from the next as-of-yet unread row of sensors in the sensor array; the set of input line buffers may be implemented in a “circular” nature.
The various output pixels in a row of the reconstructed image will most likely map back to input pixels that are on different rows of the sensor array (this is due to the distortion). For example, a first output pixel in a particular row of the reconstructed image may map back to an input pixel that is in one row of the sensor array, while another output pixel in the same particular row of the reconstructed image may map back to an input pixel that is in a different row of the sensor array. Thus, the various output pixels in a particular row of the reconstructed image will likely map back to input pixels that span across multiple rows of the sensor array. This means that in order to determine the output pixel values for all of the output pixels in that particular row of the reconstructed image, it will be necessary to refer to input pixel values on multiple rows of the sensor array. These rows of input pixel values are stored in the input line buffers; thus, there needs to be enough input line buffers to ensure that enough rows of input pixel values are stored to enable all of the output pixel values in any row of the reconstructed image to be determined. For some rows of the reconstructed image, the backward interpolation technique may need to refer to relatively few rows of input pixel values (and hence, may need relatively few input line buffers). For other rows of the reconstructed image, however, the backward interpolation technique may need to refer to many rows of input pixel values (and hence, may need many input line buffers). Enough input line buffers need to be provided to cover the worst case scenario to ensure that, for any row of the reconstructed image, all output pixel values for that row can be determined. This could be a large number of input line buffers.
As the number of input line buffers increases, the amount of storage required to perform the backward interpolation technique increases proportionally, as does the expense of the digital camera that uses the backward interpolation technique. To minimize cost, and potentially for other reasons as well, it is desirable to keep the number of line buffers in an image processing system to a minimum.
One additional drawback of the backward interpolation technique derives from the random manner in which it samples from the input image. If pixels in the output image are to be filled in a consecutive manner, and if the input image is distorted, then the regions of the input image that are sampled to determine the values of the output image often will not be adjacent to each other in the input image; the sampling window will tend to “jump around.” Because there is no guarantee, using the backward interpolation technique, that a (e.g., 4×4) window of input pixels sampled to produce a first output pixel will overlap at all with another window of input pixels samples to produce a second output pixel adjacent to the first output pixel in the reconstructed image, there is no opportunity for re-use of sampled input values from output pixel to output pixel. When a separate fresh window of input pixels needs to be completely resampled for each subsequent output pixel to be filled, bandwidth requirements in the device using the backward interpolation technique increase, as a larger quantity of sampled values need to be communicated for each and every output pixel. Additionally, when the input image is a mosaic image, the backward interpolation technique may require each such windows of sampled input pixels to be separately de-mosaiced prior to the computation of the value of the corresponding output pixel. This repetitive de-mosaicing, which cannot take advantage of window overlap that often does not exist, may require an inordinate amount of computational resources.